Skip to content

Fix pg_upgrade compatibility: omit removed catalog columns from views (re-release 0.2.2)#18

Merged
jnasbyupgrade merged 32 commits into
Postgres-Extensions:masterfrom
jnasbyupgrade:binary-pg-upgrade-ci
May 1, 2026
Merged

Fix pg_upgrade compatibility: omit removed catalog columns from views (re-release 0.2.2)#18
jnasbyupgrade merged 32 commits into
Postgres-Extensions:masterfrom
jnasbyupgrade:binary-pg-upgrade-ci

Conversation

@jnasbyupgrade

@jnasbyupgrade jnasbyupgrade commented Apr 30, 2026

Copy link
Copy Markdown
Contributor

Re-release of 0.2.2 (originally released 2026-04-27) fixing pg_upgrade
compatibility with PostgreSQL 12 and 17+.

pg_upgrade physically copies data files and re-applies schema definitions on
the new cluster. Any view referencing a catalog column removed in the new
PostgreSQL version will cause the upgrade to fail. The initial 0.2.2 release
missed several such columns:

  • relhasoids — removed from pg_catalog.pg_class in PG12
  • relhaspkey — removed from pg_catalog.pg_class in PG17
  • attcacheoff — removed from pg_catalog.pg_attribute in PG17

Those columns are removed in this release.

Also, the existing pg-upgrade-test CI job used pg_upgradecluster (dump/restore),
which reinstalls extensions at the current version and masks exactly this class
of problem. Replaced with binary pg_upgrade, which physically copies data
files and correctly catches view/column incompatibilities across major versions.

jnasbyupgrade and others added 22 commits April 30, 2026 17:53
…test

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
`relhasoids` was removed in PG12. When binary pg_upgrade restores view
definitions on PG12, `_cat_tools.pg_class_v` fails because it references
`c.relhasoids` (included by the dynamic column-list builder on PG10/11
where the column exists).

Explicitly add `relhasoids` to the omit list in `omit_column()` so the
view is defined without it on all PostgreSQL versions, making binary
pg_upgrade from PG10/PG11 to PG12+ work correctly.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
pg_upgrade writes multiple log files (pg_upgrade_internal.log,
pg_upgrade_server.log, pg_upgrade_dump_*.log, etc.) to its working
directory. Run it from a dedicated /tmp/pg_upgrade_logs dir and use
`tail -n +1` on failure to print all logs with their filenames.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Pass --no-data-checksums to pg_createcluster so the new cluster
  matches the old cluster (which pg-start creates without checksums)
- Use find+xargs instead of bare glob for log output so it doesn't
  fail when pg_upgrade exits before writing any log files

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Only run on push to master (not PR branches); pull_request handles
  those, preventing each PR push from triggering two CI runs
- Detect old cluster's data_checksums setting and pass --data-checksums
  to pg_createcluster only if needed, instead of --no-data-checksums
  which doesn't exist pre-PG18

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The psql query to detect checksums was running after pg_ctlcluster stop,
so it always failed silently and checksum_flag was never set.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
PG18 changed initdb's default to enable data checksums. Use
pg_controldata (works on stopped clusters) to detect the old cluster's
checksum setting, then:
- If old cluster has checksums: pass --data-checksums to new initdb
- If not, and new initdb supports --no-data-checksums (PG18+):
  explicitly disable to override the new default
- Otherwise: no flag needed (pre-PG18 default is no checksums)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Instead of detecting and matching checksums, recreate the old cluster
with --data-checksums immediately after pg-start, and always pass
--data-checksums when creating the new cluster. Both clusters
consistently have checksums; no detection logic needed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Add pg_isready -t 30 after pg_ctlcluster start to wait for the
  socket to be available before proceeding
- Remove the standard 'test' job (not needed on this branch)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
pg_createcluster defaults to peer/md5 auth; the container runs as root
so peer auth fails for user postgres. Pass --auth trust to match what
pg-start configures.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Like relhasoids (removed in PG12), relhaspkey was removed in PG11.
Without excluding it, binary pg_upgrade from PG10 to PG11+ fails with
"column c.relhaspkey does not exist".

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Set PGUSER=postgres at job level, eliminating -U postgres / PGUSER=postgres
  throughout all steps
- Set INITDB_OPTS=--data-checksums --auth trust at job level so both
  pg_createcluster calls stay in sync
- Fix log capture on failure: PG17+ writes pg_upgrade logs to
  $new_datadir/pg_upgrade_output.d/ instead of CWD; search both paths

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
attcacheoff was an internal column in pg_attribute removed in PG18.
Without excluding it, binary pg_upgrade to PG18 fails with
"column a.attcacheoff does not exist".

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…_CONFIG

- PGUSER: postgres moved to top-level env (applies to all jobs)
- PGDATABASE set at step level where a specific db is needed
- PG_CONFIG set at step level for new cluster install (make inherits env)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@jnasbyupgrade jnasbyupgrade changed the title ci: use binary pg_upgrade instead of pg_upgradecluster Fix pg_upgrade compatibility: omit removed catalog columns from views (re-release 0.2.2) May 1, 2026
jnasbyupgrade and others added 7 commits May 1, 2026 14:35
README: warn that catalog-exposing objects are not a stable API

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Remove step-level env: blocks to avoid future duplication.
PG_CONFIG is now at pg-upgrade-test job level alongside INITDB_OPTS.
PGDATABASE values are inlined with -d flags since they differ between
steps and a job-level value would affect make test calls.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
HISTORY: remove re-release note (now in PR description only)

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ate test

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…vel env

Job-level PG_CONFIG broke the old cluster install since the new PG
isn't installed yet at that point. Pass it inline as a make argument
for the new cluster step instead.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
jnasbyupgrade and others added 3 commits May 1, 2026 15:02
…ster

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
paths-ignore skips CI entirely when only *.md or *.asc files change.

all-checks-passed is a single stable check name suitable for use as a
required status check in branch protection rules. It fails if any job
failed or was cancelled, passes if all passed or were skipped (e.g. on
a docs-only push). It also self-inspects ci.yml to catch any jobs
accidentally omitted from its needs list.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@jnasbyupgrade
jnasbyupgrade merged commit d203d88 into Postgres-Extensions:master May 1, 2026
17 checks passed
@jnasbyupgrade
jnasbyupgrade deleted the binary-pg-upgrade-ci branch May 1, 2026 20:21
jnasbyupgrade added a commit that referenced this pull request May 1, 2026
… (re-release 0.2.2) (#18)

Re-release of 0.2.2 (originally released 2026-04-27) fixing `pg_upgrade`
compatibility with PostgreSQL 12 and 17+.

`pg_upgrade` physically copies data files and re-applies schema
definitions on
the new cluster. Any view referencing a catalog column removed in the
new
PostgreSQL version will cause the upgrade to fail. The initial 0.2.2
release
missed several such columns:

- `relhasoids` — removed from `pg_catalog.pg_class` in PG12
- `relhaspkey` — removed from `pg_catalog.pg_class` in PG17
- `attcacheoff` — removed from `pg_catalog.pg_attribute` in PG17

Those columns are removed in this release.

Also, the existing `pg-upgrade-test` CI job used `pg_upgradecluster`
(dump/restore),
which reinstalls extensions at the current version and masks exactly
this class
of problem. Replaced with binary `pg_upgrade`, which physically copies
data
files and correctly catches view/column incompatibilities across major
versions.

---------

Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
jnasbyupgrade added a commit to jnasbyupgrade/cat_tools that referenced this pull request Jul 15, 2026
eb84bc6 Stamp 2.1.0
9fbe6f4 Fix results ordering, control file whitespace, ENABLE_* override, debug levels (Postgres-Extensions#31)
88bb4f2 Add Claude Code GitHub Actions workflows (Postgres-Extensions#41)
af5bbbb ci: pass repo owners to run-tests.yml for fork-account matching (Postgres-Extensions#40)
c7928af Fix repo-root guard to work inside a git worktree (Postgres-Extensions#39)
b062fca ci: point reusable test workflow at @master
b6cdbfd Add CI workflows and multi-session PR guard (Postgres-Extensions#33)
1ba0987 Stamp 2.0.3
1931cbe Fix pgxntool-sync remote and make it runnable without make (Postgres-Extensions#37)
8176304 Stamp 2.0.2
3e142ab Fix parse_control_file: remove comments before stripping quotes (Postgres-Extensions#27)
cacc301 Stamp 2.0.1
bf1db6b Fix bash 3.2 / Linux compatibility issues (Postgres-Extensions#26)
62d0fcb Fix broken ifeq for --load-language=plpgsql on PG < 13 (Postgres-Extensions#24)
121f0b3 Stamp 2.0.0
ad3ca7e Remove .source support; add test/install, test/build, and verify-results (Postgres-Extensions#18)
c010cf8 Fix bash 3.2 compatibility (Postgres-Extensions#23)
abeb9d3 Remove .source file support from pg_regress integration (Postgres-Extensions#22)
08c1879 Stamp 1.1.2
6e0dad2 Fix double --dbname bug that defeated unique test database names
639756c Stamp 1.1.1
6ba3176 Fix pg_tle exception handler and empty upgrade files (Postgres-Extensions#15)
3b8cb2a Stamp 1.1.0
550a901 Remove commit.md (maintained in pgxntool-test)
d73ca93 Add unique test database names to prevent conflicts (Postgres-Extensions#13)
9b344be Add update-setup-files.sh for 3-way merging after pgxntool-sync (#12)
ab7f6e2 Stamp 1.0.0
3a571ba Add pg_tle support and modernize test infrastructure (#11)
b96ea6d Add support for Claude code; build and doc improvements (#9)
e9c24de Fix pg_regress on versions > 12 (#5)

git-subtree-dir: pgxntool
git-subtree-split: eb84bc6e87e21f2ced11ff1b8ddb4028b7c67c8f
jnasbyupgrade added a commit to jnasbyupgrade/cat_tools that referenced this pull request Jul 15, 2026
…de scripts (fixes Postgres-Extensions#28)

The machinery that generates our versioned SQL (the cat_tools--X.Y.Z.sql
install scripts and cat_tools--A.B.C--X.Y.Z.sql update scripts) from .sql.in
sources, and the DATA list that installs it, is moved out of the Makefile into
a new, documented sql.mk (included after pgxntool/base.mk so it can use
EXTENSION_VERSION_FILES, PG_CONFIG, MAJORVER, datadir, ...).

sql.mk documents the GNU Make two-phase (parse vs. recipe) hazard at the root
of Postgres-Extensions#28: we GENERATE the .sql we install, the generated files are gitignored and
absent on a clean tree at parse time, and any parse-time $(wildcard) over them
(including base.mk's DATA seed $(wildcard sql/*--*--*.sql)) silently comes up
short -- the cached directory listing means it never notices them later either.
The fix is to name-derive the install/update lists from the .sql.in SOURCES
(which DO exist at parse time), feed the generated names into DATA explicitly,
and $(sort) to dedup against base.mk's own wildcard. This makes the generated
upgrade scripts land in DATA reliably on a clean build, so `make install`
copies them and a later `ALTER EXTENSION cat_tools UPDATE` finds its path (Postgres-Extensions#28).

The historical cat_tools--0.1.* install list is no longer hardcoded: it is
derived by globbing the committed install-shaped .sql and subtracting the
update scripts and everything we generate from .sql.in. Globbing COMMITTED
files at parse time is safe; the rule of thumb is glob what git tracks,
name-derive what we generate.

Carry the Postgres-Extensions#18 pg_upgrade fix into the SOURCE. Postgres-Extensions#18 (re-release of 0.2.2) omitted
catalog columns that later PostgreSQL versions removed -- relhasoids (PG12),
relhaspkey (PG17), attcacheoff (PG17) -- so the stored view definitions do not
reference since-removed columns and binary pg_upgrade succeeds. That fix was
applied only to the GENERATED versioned files (sql/cat_tools--0.2.2.sql.in and
the *--0.2.2 update scripts), never to the SOURCE sql/cat_tools.sql.in. That was
harmless while the old Makefile deleted-and-did-not-reliably-regenerate the
current-version file, but this refactor regenerates it in place from the source,
which would REVERT the Postgres-Extensions#18 fix. Applying the same omit_column additions to
sql/cat_tools.sql.in makes regeneration reproduce sql/cat_tools--0.2.2.sql.in
byte-for-byte, preserving the released 0.2.2 SQL.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
jnasbyupgrade added a commit that referenced this pull request Jul 16, 2026
…#38)

PR #18 (commit 31204e1, re-release 0.2.2) fixed binary `pg_upgrade`
compatibility by omitting catalog columns that newer PostgreSQL removed
from the `pg_class_v` / `pg_attribute_v` views:

- `pg_class`: omit `oid`, `relhasoids` (removed PG12), `relhaspkey`
(removed PG17)
- `pg_attribute`: add `attcacheoff` (removed PG17) to the existing
`oid`, `attmissingval`

But #18 applied that fix **only to the generated versioned files**
(`sql/cat_tools--0.2.2.sql.in` and the `*--0.2.2` update scripts),
**never to the source** `sql/cat_tools.sql.in`. So running `make`
regenerates the current-version file from source and **silently
reverts** the #18 fix.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant